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Abstract 

One principal way to judge if the activity or program described in a practice-oriented publication is worth replicating 
is to consider the strength of its foundational component. A strong foundational component supports the credibility of 
an initiative by demonstrating that it was designed with an understanding of other similar initiatives and with 
evidence that there was a reasonable promise that the desired outcomes were realistically achievable. This paper 
reports on the findings of a mixed methods content analysis of 142 practice-oriented articles published in three 
different refereed venues between 1995 and 2009. A scale to measure the quality of the foundational component of 
the practice-oriented publications was used in the quantitative phases to test if there had been a change in the relative 
number and quality of practice-oriented publications over time. The validity of the scale was tested in subsequent 
qualitative phases of the project by an inductive examination of the characteristics of top-scoring articles. Results of 
the quantitative analysis indicated that the average quality rating of the foundational element of the articles was 
relatively low (3.37 out of a possible 6 points). The scale proved effective in distinguishing articles that were 
exemplary in other ways as well. The low overall quality score on the foundational component of practice-oriented 
publications either means that the authors of these publications are failing to provide ample documentation that these 
initiatives were evidence-based or the content analysis reveals a long-standing trend toward the adoption of 
non-evidence based practices. 

Keywords: Gender and STEM, Practice-oriented literature, Evidence-based practice 

1. Introduction 

Evidence-based practice (EBP) is an interdisciplinary approach with a long history in the health disciplines. It is an 
approach that argues policy and practice should be justified in terms of sound evidence about their likely effects. 
Currently, the term evidence-based practice is used with two different meanings. One meaning is associated with 
“best-practice.” The other meaning is associated with evidence as the basis for practice (Slavin, R. E., 2002) and 
practitioner decision-making. (Mullen, E.J., 2002) Although the discourse has been heavily shaped by its application 
to clinical settings, the parlance of evidence-based practice now extends to other fields and beyond the limitation of a 
single experimental research design. (Slavin, R. E., 2002) 

The social sciences are increasingly referring to evidence-based programs. The Society of Prevention Science created 
a set of standards of evidence that have been used by government agencies to establish lists of model programs. In 
general, evidence-based programs are based on a clearly articulated and empirically supported theory and have a 
detailed description of the intervention and measurement design with what outcomes were produced in what 
populations with what intervention. They have measureable outcomes that have been assessed with psychometrically 
strong measures, including long-term follow-ups and have been tested in a scientifically sound way with comparison 
conditions, optimally through randomized controlled studies. 

The practice-oriented literature is important in the arena of activities designed to promote interest in science, 
technology, engineering, and math (STEM) because it promotes continued innovation by making public information 
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about the design and implementation of activities and courses that strive to enhance learning and promote interest in 
careers in science and engineering. Practice-oriented publications are those that have among their principal purpose 
the intent to describe the design, delivery, and outcomes of educational activities, classes, or programs in K-12 or at 
the collegiate level. 

The purpose of the sequential mixed methods content analysis described in this paper is to provide results of a 
synthesis of the practice-oriented published STEM literature in three sources: the proceedings of the Frontiers in 
Engineering Education (FIE) and Association of Engineering Education (ASEE); and the Journal of Women, 
Minorities, in Science and Engineering(JWMSE). There were several reasons for selecting these sources. All are 
refereed and reach national and internal audiences of STEM educators. Practice-oriented articles in engineering 
education frequently cite references from these sources. 

This descriptive study examines trends over time in the relative number of practice-oriented publications in the three 
venues and considers whether a scale developed for purposes of the study points to an improvement in the quality of 
this literature over time. An improvement in the quality score over time would suggest that new practices are 
increasingly evidence based. It also suggests maturation in this distinct field of inquiry and that newer publications 
were more firmly grounded than older publication in the growing literature base. 

The rationale for using mixed methods was in part to test the validity of a scale developed for purposes of the project 
to identify exemplary practice-oriented publications that novice authors might use as models. We consider an 
inductive approach to analysis as a fundamentally qualitative approach; the deductive approach to analysis as a 
fundamentally quantitative approach. The purposes of this mixed method project fit a development design (Greene, 
Caracelli, & Graham, 1989) in that the results from the first quantitative phases of the project became the basis for 
further, extended investigation in the subsequent qualitative phases of the project that sought to demonstrate the 
validity of the scale designed to identify exemplary practice-oriented publications. 

The research questions for the mixed methods study reported in this paper are: 

1) (QUANT) Has there been a change over time (1995-2009) in the relative number of practice-oriented publications 
appearing in the three sources? 

2) (QUANT) What is the average foundational quality of practice-oriented publications? Has the foundational quality 
score improved over time (1995-2009)? 

3) (QUAL) What are additional characteristics of publications scoring the highest on the evidence-based scale? 

4) (QUANT) What evidence is there to support the validity of the scale developed to evaluate the foundational 
quality of practice-oriented publications? 

The analysis reported in this paper was conducted as part of a supplemental grant to a larger funded project (NSF 
GSE 08322913) funded by the Program for Gender Equity (PGE) at the National Science Foundation. The 
supplemental grant was designed with the goal of promoting cross-disciplinary collaboration by developing an 
accessible and comprehensive indexed database of refereed gender and STEM publications. This database makes it 
possible to conduct synthetic and longitudinal studies of research publications in gender and STEM. It also makes it 
possible to evaluate multiple dimensions of the quality of these publications in this cross disciplinary topical area in 
order to assess if the field is moving forward in that newer initiatives are grounded in a solid foundation that reflects 
knowledge of previously produced literature in the field. 

2. Literature Review 

The foregrounding of language about evidence-based practice by funders like the National Science Foundation 
reflects not only the influence of the political climate that arose around the No Child Left Behind Act of 2001, but 
also calls for accountability and assurances that proposals they funded to enhance practice were grounded in 
convincing evidence of their potential to have the promised effect on targeted outcomes. These affected not only the 
way the proposals were evaluated but also a shift to expecting a strong research and evaluation component in 
proposals for funding, even for those designed primarily to enhance practice. 

Following the enactment of the No Child Left Behind Act, the leading and largest educational professional research 
association convened a panel of experts in a variety of methodologies to develop the American Educational Research 
Association (AERA) Standards for Reporting on Empirical Social Science Research (2006). These standards apply to 
all types of social science research and research reports, including those with a practical intent. The two overarching 
principles established in the preamble to the Standards are, first, that there is adequate evidence or warrants to justify 
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the results and conclusions and, secondly, that research reports are written in such a way that they are transparent or 
explicit in the details provided about the logic of inquiry from the design through the execution of the study. 

The first part of the AERA Standards deal with what could be called the foundational element of a study or what the 
AERA Standards refer to as problem formulation. “The importance of this step in evaluating a single study deserves 
additional attention” (Leech, Dellinger, Brannagan, & Tanaka, 2010, p. 29). According to the Standards, “The 
problem formulation answers the question of why the results of the investigation would be of interest to the research 
community and how an investigation is linked to prior knowledge and research” (2006, p. 34). The foundational 
element reflects “researchers’ prior understanding of a construct and/or phenomenon under study” (Dellinger & 
Leech, 2007, p. 323). If the study concerns practice, then the Standards specify that it should be made clear “what 
those concerns are, why they are important, and how this investigation can address those concerns” (p. 34). The 
Standards go on to state that it should be clear why the intervention is appropriate for the group studied, with 
appropriate citations. 

Several reasons can be put forward for why the foundational element is a necessary condition of overall quality of a 
scholarly publication. Situating the research in the appropriate body of literature provides initial evidence of 
construct validity (Dellinger & Leech, 2007; Leech et al., 2010), establishes the credentials of the authors to conduct 
the research, and is necessary to establish the credibility of the conclusions (Beach, 2006). The foundational element 
foreshadows the conclusions by providing information about how to interpret the findings, their consequences, and 
their transferability to other settings (Leech et ah, 2010). Without a strong foundational element, its is impossible for 
authors to situate their in ways that make its contribution to knowledge evident. This is a point Beech made when he 
said: 

It is impossible to construct conclusions without relying on prior knowledge. At the same time, if the meaning 
of research findings is dictated solely by prior knowledge, then we cannot learn from new studies. The 
problems that researchers face, therefore, is not one of how to put aside prior knowledge, but rather one of how 
to capitalize on prior knowledge and use it to extract as much new knowledge (italics his) as possible from the 
findings ” (Beach, 2006, p. 502). 

A strong foundation makes it possible to pinpoint exactly the contribution of the results and conclusions to the body 
of knowledge. 

While it can be said that a strong foundational element is a requirement for an article to merit the designation of 
being exemplary, it is not the only indicator of quality that is critical to an overall assessment of quality. Other 
elements, such as the execution of the research and the quality of inferences are part of a comprehensive framework 
to evaluate the overall quality of research reports (Dellinger & Leech, 2007; Leech et al., 2010; O’Cathian, 2010). A 
report about practice does not merit replication, and thus be worthy of vetting to a larger audience, without a strong 
foundational element that signals that the initiative was evidence based. 

3. Methods 

Following the creation of the database of gender and STEM publications (N=612), this mixed methods content 
analysis was conducted in six steps. 

1) A database of gender and STEM articles was coded in NVIVO10 using an initial, deductively derived 10-item 
coding scheme. One of the codes distinguished practice-oriented articles from research and conceptual articles. 

2) A pilot study was conducted with 25 practice-oriented publications to gauge the effectiveness of a scale that was 
developed for purposes of the study to measure the foundational quality of the publications. 

3) The remaining practice-oriented publications were evaluated using the foundational scale. 

4) During the pilot study, one top-scoring article was coded using an emergent approach and three additional 
characteristics of quality were identified. 

5) All the top-scoring articles were coded to determine if the additional quality characteristics were present. 

More details about the different phases of the research project are described in the next section. 

3.1 Creation of the Database of Practice-Oriented Publications 

A database of all gender related publications appearing in three sources (FIE and ASEE proceedings, JWMSE) was 
created and coded in NVIVO10 software. The database was assembled by conducting electronic searches of the 
literature in four sources to identify publications that used the search terms [gender or female or girls or women or 
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gender] and [science or engineering or technology] in the title. All publications from the JWMSE and those meeting 
the search terms from 1995-2009 were added to the database (N=612). 

All of the articles were initially coded for ten different variables. These variables were: three keywords; if it 
acknowledged support from funding; if the sample included members of other under-represented groups and, if so, 
what groups, in addition to women; the nature of the sample (e.g. K-12 students, K-12 teachers or other personnel, 
undergraduate students, community college students, etc.); discipline or field of focus; if it was single or 
multi-institutional; publication year and source; type of publication (research, practice, literature review, theoretical 
or conceptual). 

The focus of this paper is on the sub-set of articles identified to be practice-oriented (N=142). Practice-oriented 
publications were identified as those that that had among their principal purpose the intent of describing the design, 
delivery, and outcomes of educational activities, classes, or programs in K-12 or at the collegiate level. 
Research-based and practice-oriented publications were distinguished in terms of the stated purpose of the 
publication. These two types of publications also tended to differ in the presentation and analysis of data. Most 
practice-oriented articles contained little or no data. 

3.2 Pilot Study 

An initial pilot study was conducted with a random sample of practice-oriented publications (n=25) from all three 
sources in order to develop a set of operational decision rules about how to apply a 6-point ordinal scale developed 
by the lead author to assess the extent that the documentation presented in each article reflected a grounding in 
evidence. Raters focused on explicit language provided in the text. Raters applied a value of “0” to articles with no 
supporting related references; “1” to those with one related reference; “2” to those with two related references, and 
“3” to those with three or more related references. Articles scored with a “4" where those with three or more relevant 
references and the references to the literature included implicit or explicit references that indicated that data to 
support the effectiveness of the intervention was considered before the decision was made to initiate the intervention. 
A “5” on the evidence-based scale indicated three solid references plus the presence of explicit language that the 
intervention was a scale-up of a smaller project or pilot study. The highest score, “6”, was applied by the scorers 
when evidence was supplied in the publication that the practice derived from a theoretical model or framework. 
Raters exercised some judgment in determining which references to include in the count. Google and Wikipedia 
references, for example, were not included. 

During the pilot study, three readers used the 6-point foundational scale to score 25 articles. They continued meeting 
and comparing scores until three-way inter-rater reliability was consistently in excess of 90%. The satisfactory level 
of inter-rater reliability was taken as one indication that a set of operational rules for coding was being used 
systematically. Preliminary frequency counts supported continued use of a 6-point scale. The second and third author 
went on to score all the remaining practice-oriented publications. 

3.3 Inductive Analysis of High Scoring Articles 

A qualitative or inductive approach was used by the second and third of the authors in the third phase and final phase 
of the study. They used an open or inductive approach to identify additional characteristics of the single article that 
was scored a value of 6 on the 6-point scale. This was the only practice-oriented publication that met all of the 
criteria identified in the scale. 

4. Results 

The results reported in this paper are from an analysis of the body of literature of practice-oriented publications 
appearing in three sources often accessed by engineering educators (ASEE, FIE, and JWMSE). This type of 
publication accounted for 142 of 612 (23.2%) of the gender-related publications appearing in the three sources 
between 1995 and 2009. Most of these articles described either formal or informal activities or courses designed to 
promote the recruitment or retention of students in STEM at the K-12 or undergraduate level. Very few articles 
described a program or activity involving graduate students or faculty members. 

In the following section, we describe trends over time and by source in the relative number of practice-oriented 
publications. In addition, we report on what the evidence-based foundational scale tells us about the quality of these 
publications and the extent they were grounded in a body of evidence. Finally, we report on the findings from the 
qualitative analysis of additional qualities of exemplary publications. 
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4.1 Research Question 1: Number of Practice-Oriented Publications by Time Frame and Source 

The first table displays the total number of gender-related practice-oriented publications appearing in the three 
sources for each of three time periods between 1995 and 2009. The numbers in some of the cells are too small to 
warrant the use of statistical analysis. More than 20% (142 of 612) of all the publications appearing in these three 
sources between 1995 and 2009 were practice-oriented. Overall, practice-oriented publications were the highest 
percent of all publications appearing in the ASEE proceedings (n=90, 32.6%), followed by the proceedings from the 
FIE conferences (n=18, 30%). Practice-oriented publications were the smallest percentage of all articles appearing in 
JWMSE (n=34, 12.2%). 


Table 1. Number of practice-oriented articles by time frame and source 


Time Frame 

ASEE 

Practice 

Total 

Percent 

1995-1999 

11 

27 

40.74 

2000-2004 

40 

100 

40.00 

2005-2009 

39 

149 

26.17 

Total 

90 

276 

32.61 


FIE 



1995-1999 

0 

0 

0.00 

2000-2004 

14 

26 

53.85 

2005-2009 

4 

34 

11.77 

Total 

18 

60 

30.00 


JWMSE 



1995-1999 

5 

88 

5.68 

2000-2004 

15 

86 

17.44 

2005-2009 

14 

102 

13.73 

Total 

34 

276 

12.32 


We hypothesized that there would be a decline in the number of practice-oriented publications in the various venues 
over time as the disciplines matured and the research base became more robust. This hypothesis was supported in 
that percentage of all publications that were practice- oriented declined in each of the three venues for the last time 
period (2005-2009).The percentage dropped from 40% to 26.17% for ASEE publications during that time period and 
from 53.8% to 11.7% for FIE. The decline was not so dramatic for publications appearing in JWMSE. 

4.2 Research Question 2: The Quality of the Foundational Element of the Practice-Oriented Literature 

Table 2 provides the results of the measure of foundational quality used for this paper by showing the number of 
practice-oriented articles that were scored at each point on the 6-point ordinal scale. The higher the score on the scale, 
the stronger the foundational materials in that they provided more evidence from the literature to justify the choice of 
intervention or activity described in their publication. Low scores signal that authors failed to provide documentation 
that their decision to initiate a new activity, course, or program grew out of knowledge of other initiatives with 
similar goals that had been proven successful. 
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Table 2. Frequency of the quality scale of the foundational element of practice oriented publications 


Practice 

Articles 


Score 

Frequency 

Percentage 

0 

10 

7.04 

1 

5 

3.52 

2 

8 

5.63 

3 

44 

30.99 

4 

50 

35.21 

5 

24 

16.90 

6 

1 

0.70 

Total 

142 

100.00 


Overall, the scores on the evidence-based scale indicate that the quality of the practice-oriented literature is not high. 

Results of a one-way analysis of variance (ANOVA) considering differences in quality scores across time frame, 
disputed the hypotheses that the quality of practice-oriented publications has improved over time (F=1.490; p=,229). 

The overall mean on the scale designed to measure the foundational grounding of the article was 3.37out of a 
possible 6 points. This indicates that most publications provided three or fewer references to justify the selection of 
an activity or program. About one-tenth of the articles either made no reference to literature supporting the choice of 
the activity, or provided only one reference. 

On the other hand, 25 twenty-five of 142 publications (17.6%) scored either a 5 or 6 on the scale. The authors of 
these publications provided stronger documentation for the activity they chose by including explicit references to a 
pilot or scale up study or by describing the grounding of the activity in a theoretical or conceptual model or 
framework. Other characteristics of these articles that might warrant the designation of exemplary are discussed in 
the next section. 

4.3 Research Question 3: Additional Characteristics of Exemplary> Evidence-Based Practice-Oriented Publications 

Our third research question required the use of a qualitative or inductive approach to identify additional 
characteristics of the design and execution of practice-oriented publications that scored high on our foundational 
quality scale. Only one article surfaced during the pilot study where three raters independently nominated as 
exemplary in an overall sense. This was the only article of 124 to be scored by all raters as a 6 on 6-point scale. This 
was an article by Jessup, Sumner, and Barker appearing in JWMSE in 2005 and titled, Report form the Trenches: 
Implementing Curriculum to Promote the Participation of Women in Computer Science. The article describes the 
design and implementation of an undergraduate computer science course at the University of Colorado at Boulder in 
which students work with local community service agencies to build computational solutions for problems the 
agencies confront. 

The first quality beyond the foundational component that distinguished the Jessup et al. (2005) publication is that the 
authors built credibility early in the publication in the review of the literature. Specifically, the authors cited 
recommendations from other successful programs and then explained how they would implement these 
recommendations. By establishing this strong credibility early in the review, the authors generated interest in the 
program’s methods, and, more importantly, they developed a stronger case for the program’s purpose. The authors 
explained: 

Several types of solutions are proposed in the literature to attract and retain women in computing study ... we have 
borrowed from some of the recommendations above, as well as from the more general literature on women’s 
preferences, in building our course (p. 275). 
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The second exemplary feature of Report from the Trenches (Jessup, E., Sumner T., Barker, L., 2005) is Jessup et al. 
(2005) explicitly identified and described the theory TC was grounded in. The authors provided an explanation that 
facilitated understanding of the paradigm and proceeded with a strong rationale for the application of the theory. 
Moreover, Jessup et al. (2005) subsequently provided documentation of how the theory was integrated into the course 
they designed. By describing how designed-based learning might be observed, the authors created an opportunity for 
the reader to understand the theory in context of the program: 

Thus, in design-based learning, skill and knowledge development activities are all situated within an authentic 
problem-solving context in which success is measured by the ability to produce a useful and useable artifact... 
Design-based learning can counteract impersonalization and humanize both the classroom setting and the 
technology-development process. Within the TC course, the projects are more complex than any student can take 
on individually. By necessity, all work must be performed by teams of students working together intensely, both 
within and outside of the formal class period. Students within the teams learn to depend on each other’s unique 
skills and expertise (p. 276). 

The third quality characteristic is the way the authors continually integrated the specified problem and literature 
surrounding the problem into the design of the course. Instead of only identifying the problem as a failure to attract and 
retain women in computing, Jessup et al. (2005) identified specific causes of the lack of women, including a lack of 
confidence, lack of mentoring, and the impersonal feel to the coursework and then explained how these causes were 
factored into the design of TC. Specifically, the authors discussed how participants’ preference for teamwork led to the 
decision to have women work in groups, and that through this decision, the program negated a confidence loss more 
commonly experienced in women than in men. It also gave women a chance to form peer mentors. Most importantly, 
the authors described how working with local community members created a humanizing feel, which research shows 
women desire in a career (Jessup et al. 2005, pp. 274-276). 

Preliminary support for the validity of evidence-bases scaled developed for this publication comes from the fact that 
the JWMSE publication, Report From the Trenches by Jessup et al. (2005), scored as exemplary using both a 
quantitative index and through additional qualitative analysis. Subsequent analysis during the qualitative phase of the 
mixed methods study confirmed the initial score of the publication as exemplary. Inductive coding of other qualities of 
the publication pointed to three additional qualities that warrant the designation as exemplary, all of which are related 
to what is traditionally found in an introduction or statement of the problem: (1) documentation was provided about 
how the course was designed based on recommendations from reports of other programs that had been proven to be 
successful, (2) the authors employed a theoretical framework, and (3) an explicit link was made between a problem and 
the solution. 

4.4 Research Question 4: Additional Support for the Validity of the Scale Developed to Measure the Foundational 
Quality of Practice Oriented Publications 

In the final step of the analysis, we returned to a quantitative or deductive approach to collect additional evidence 
that the foundational scale we developed was effective in distinguishing exemplary from non-exemplary publications. 
After using an inductive approach to identify additional characteristics of a practice-oriented article ranked as 
exemplary in overall quality by three raters independently, we sought to see if some of the same qualities would 
extend to other articles scoring high on the foundational scale. Confirmation of this hypothesis would support the 
validity of the scale and mean that we had moved closer to developing a way to measure the quality of 
practice-oriented publications. 

To further test the effectiveness of the foundational scale in distinguishing high from low quality practice-oriented 
publications, the second and third author re-read all of the publications scoring a 5 or 6 on the foundational scale 
(n=24) and coded them for the absence or presence of each of the three additional dimensions of quality identified 
during the qualitative phase. All of these were part of what could be called problem formulation. The three qualities 
were: (1) documentation was provided about how the course was designed based on recommendations from reports of 
other programs that had been proven to be successful, (2) the authors employed a theoretical framework, and (3) an 
explicit link was made between a problem and the solution. Table 3 shows the results of this analysis. 
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Table 3. Presence of additional quality characteristics in top-scoring practice-oriented publications 


Article # 

Article ID Number 

Quality 1 

Quality 2 

Quality 3 

1 

001JWMSE99 

X 

X 

X 

2 

013JWMSE00 


X 

X 

3 

012JWMSE01 

X 

X 


4 

010JWMSE03 

X 


X 

5 

009JWMSE03 

X 

X 

X 

6 

006JWMSE03 



X 

7 

016JWMSE03 


X 

X 

8 

016JWMSE05 

X 

X 

X 

9 

007JWMSE07 

X 


X 

10 

007JWMSE09 

X 


X 

11 

013JWMSE09 

X 

X 

X 

12 

025_ASEE03 

X 


X 

13 

013_ASEE04 

X 

X 

X 

14 

019_ASEE04 

X 

X 

X 

15 

022_ASEE04 


X 


16 

024_ASEE04 


X 

X 

17 

009_ASEE05 



X 

18 

008_ASEE05 

X 

X 

X 

19 

019_ASEE06 



X 

20 

022_ASEE07 

X 

X 

X 

21 

028_ASEE07 



X 

22 

012_ASEE07 

X 


X 

23 

034_ASEE07 


X 

X 

24 

012_ASEE08 

X 


X 


The results shown in Table 3 demonstrate that the 24 articles scoring high on the foundational scale also generally 
shared three additional characteristics. Fifteen of 24 had described their projects as emerging from a pilot study; 14 
of 24 described a theoretical foundation for their initiative; and 22 of 24 traced an explicit connection between the 
way the intervention was designed and their beliefs about the root causes of the historically long-standing 
under-representation of women in STEM fields. 

5. Discussion and Conclusions 

The analysis reported here is part of a larger project designed to identify trends over time in publications about 
gender and STEM. The focus of the analysis in this stage of the project was on practice-oriented publications. The 
principal overall conclusion from the analysis emerging from the first phase of this larger project is that majority of 
authors of practice-oriented articles failed to document that the decision made to implement a new program or 
activity was based on evidence. There was no convincing evidence that the quality of these publications has 
improved since 1995. This suggests that while well-meaning, many interventions designed to promote women’s 
interest in STEM fields were launched without a strong foundation or understanding of the outcomes of similar 
initiatives in other settings. 

We argue that practice-oriented publications and research publications have different purposes and, consequently, 
require a different set of standards to evaluate. Practice-oriented articles that appear in refereed venues are implicitly 
or explicitly put forward for replication as a way to design activities and programs that promote improvement in 
institutional practices and policies. While strong evaluations of the interventions can be hoped for, this kind of data 
does not lend itself to the more sophisticated analysis that would be expected of a research article appearing in a 
premiere outlet. 
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Open coding of a set of 24 articles scoring the highest on the foundational scale, demonstrates that these types of 
publications have their own set of characteristics that distinguish those that are exemplary that are quite distinct from 
what would be expected of a research article. The juxtaposition of the additional characteristics with the score on the 
foundational scale support the validity of the scale, particularly when used with other qualities that shape problem 
formulation. These criteria might be useful by editors and reviewers of practice-oriented publications. The list of 
qualities also can be helpful to novice researchers as they try to document the effectiveness of interventions they 
have designed. There is not reason they would not be generalizable to a non-STEM context. 

One of the goals of the larger project is to use mixed methods to analyze quality dimensions of the literature that has 
been published on topics related to gender and interest and success in STEM majors and careers to see if there is a 
detectable shift toward more robust research designs and more evidence-based practice. The scale used in this 
research was developed to measure the extent that practice-oriented publications were evidence-based, in that they 
described activities or programs developed with a solid foundation of evidence from the literature about similar or 
related initiatives that had been proven to be effective. The argument for why this is important is that an initiative 
does not justify replication in other settings if it was not designed with a solid grounding in evidence. A second 
reason is that it is impossible to identify the contribution to knowledge or construct conclusions without relying on 
prior knowledge. (Beach, K. D., Becker, B. J., Kennedy, J. M., 2007) Programs or activities can have innovative 
components, even as they build on knowledge of related activities that have been proven effective. 

The assessment of the quality of the foundational element of practice-oriented publications reported here offers a 
number of insights to scholars and practitioners setting out to write high-quality publications to showcase their 
efforts. The research reported here underscores the importance of being explicit about framing the publication in 
ways that document the source of the idea and in being explicit about linking the initiative to publications that 
preceded it. Would-be authors will enhance the credibility of their practice-oriented publications by doing the 
following: 

1) Explicitly identify the source of the problem and its connection to the solution in theory and practice. 

2) Describe how the program was designed with knowledge derived from previous research and explain how the 
program applied recommendations adopted from other programs with demonstrated effectiveness. 

3) Detail broad and potential implications to practice and policy (e.g. course content, curriculum design). 

4) Identify limitations and explain in what settings the intervention might not be effective. 

The implications of relatively low average quality scores across the body of practice-oriented literature are 
worrisome for several reasons. One reason for this is that it is an indication that authors of practice-oriented 
publications often did not document that the selection of an activity was solidly grounded in knowledge that the 
intervention had the potential to be effective. A second reason is that it suggests that relevant literature, though 
readily accessible through electronic means, is not being incorporated in the design of new programs as frequently as 
would be expected. One repercussion of both of these to broadening participation in STEM is that it creates 
redundancy between publications. A second is that it suggests that there has been a significant investment of time, 
energy, and resources in perpetuating ideas that have not been documented to be effective in practice. 
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